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Abstract.  Our  group  has  participated  into  Relevance  Feedback  (RF)  Track  in  TREC2008.  In  our 
experiments,  two  kinds  of  techniques,  query  expansion  and  search  result  re-ranking  based  on 
document  relevance  model,  are  adopted  to  improve  the  retrieval  performance.  The  TMiner  search 
engine,  from  IR  group  of  Tsinghua  University,  is  used  as  our  text  retrieval  system. 

1,  Introduction 

Tsinghua  University  Information  Retrieval  Group  (THUIR)  has  participated  into  the  first 
Relevance  Feedback  Track  of  TREC2008.  The  TMiner  search  engine  has  been  used  as  our  text 
retrieval  system,  because  the  processing  capability  and  flexibility  of  this  system  on  large  text  data 
has  been  testified  during  many  years’  Web  Track  and  Terabyte  Track.  In  the  track,  we  studied  two 
approaches:  I)  query  expansion,  2)  search  result  re-ranking  based  on  document  relevance  model. 
Query  Expansion:  Terms  in  the  annotated  documents  (feedback)  are  used  to  expand  the  original 
query;  the  new  bom  queries  are  sent  to  the  search  engine  for  further  information  retrieval;  users 
get  the  documents  retrieved  by  the  expanded  queries. 

Search  Result  Re-ranking:  The  relevance  between  the  annotated  documents  and  other  documents 
are  used  to  influence  the  search  results;  users  finally  get  the  re-ranked  document  list.  In  detail,  we 
have  experimented  two  different  methods  on  which  search  result  re-ranking  based:  a)  Clustering;  b) 
Documents  Relevance  Model. 

The  rest  of  this  paper  is  stmctured  as  follows.  After  the  introduction  of  Query  Expansion  approach 
in  Section  2,  Search  Result  Re-ranking  is  discussed  in  Section  3.  The  evaluation  results  of  the 
submitted  rans  are  illustrated  in  Section  4.  The  last  section  contains  summaries  and  outlines 
promising  future  work. 

2,  Query  Expansion 

In  Query  Expansion  approach.  Terms  in  the  annotated  documents  (feedback)  are  used  to  expand 
the  original  query;  the  new  bom  queries  are  sent  to  the  search  engine  for  further  information 
retrieval.  Two-phase  process  has  been  adopted:  a)  Term  selection  with  feedback  documents;  b) 
Search  Result  Integration. 

2.1  Term  Selection 

We  selected  expansion  terms  using  local  context  analysis  method.  Assume  the  query  to  be 
expanded  is  Q,  the  query  terms  in  Q  are  qi,  q2...qm,  the  collection  being  searched  is  C  and  the  set 
of  relevant  documents  is  S.  Our  approach  prefers  terms  of  higher  co-occurrence  with  query  terms. 
Specifically,  we  will  derive  a  function  y(t,  Q)  which  measures  how  good  a  term  t  is  for  expanding 
based  on  t’s  co-occurrence  with  q,  in  S.  All  terms  are  ranked  byf  and  the  best  k  terms  are  selected. 
We  have  adopted  the  following  formula  [  1  ]  to  measure  the  degree  of  co-occurrence  of  t  with  q,: 
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co_degree(,t,qi)  =  +  1)  Xlog  (tf^qi.d)  +  1> 

des 

where  tf{t,  d)and  tf(qi,  c?)are  the  frequencies  of  t  and  q,  in  document  d,  respectively. 

To  combine  the  degrees  of  co-occurrence  with  all  query  terms,  the  following  function  is  used: 

where  ic?/(t)  is  the  inverse  document  frequency  of  t  in  the  whole  collection  C.  so  is  idf(qi). 


2.2  Search  Result  Integration 


Intuitively,  terms  selected  from  the  relevant  documents  should  be  treated  separately  with 
documents  selected  from  irrelevant  documents.  We  first  tried  the  following  formula,  in  which 
terms  from  different  relevant  documents  are  treated  as  if  they  are  from  one  document.  This  is 
because  we  suppose  relevant  documents  are  prone  to  be  concept  focused,  and  on  the  contrary  the 
irrelevant  documents  may  share  many  unique  concepts  respectively. 


sim(Q,  D)  =  sim(Q,  D)  +  a  *  sim 


.D) 


However,  through  our  training  experiments,  the  irrelevant  part  sim(Dneg  ,  D)  doesn’t 

help,  so  it  was  finally  abandoned.  As  a  result,  the  actual  formula  is  adopted. 


sim(Q,  D)  =  sim(Q,  D)  -I-  a  *  sim 


Figure  1  shows  the  MAP  changes  with  the  number  of  the  selected  terms.  Different  curves  are  the 
results  with  different  combination  parameter  a. 


QE(D) 


#(QE)  terms 


Figure  1 :  MAP  with  the  increase  of  the  selected  terms  under  different  combining  parameter:  a 


3,  Search  Result  Re-ranking 

We  have  studied  two  re-ranking  methods  to  improve  the  retrieval  performance  using  relevance 
feedback  information:  clustering  and  document  relevance  model. 


3.1  Clustering 

Under  the  assumption  that  relevant  documents  share  several  concepts  (needs  behind  the  query), 
our  intuitions  are:  by  appropriate  clustering  algorithms,  1)  relevant  and  irrelevant  documents 


would  be  clustered  into  several  different  clusters  (R-cluster  and  NR-cluster)  separately;  2)  There 
would  be  more  NR-clusters  than  R-clusters.  If  several  feedback  relevant  documents  are  in  one 
cluster,  there  is  high  probability  that  a)  it  is  a  R-cluster,  b)  other  documents  in  the  cluster  are  also 
relevant.  The  known  irrelevant  documents,  on  the  contrary,  can  be  used  to  determine  NR-clusters. 
Unfortunately,  the  experimental  result  is  disappointing.  Although  we  used  all  clustering  methods 
of  CLUTO  [2],  relevant  documents  cannot  be  clustered  with  high  purity.  Figure  2  shows  the 
clustering  results  of  bagglo-wclink-10[2]  parameter. 


Figure  2:  The  clustering  result  using  CLUTO 

The  “rel/all  rel”  series  means  the  number  of  relevant  docs  divide  the  number  of  all  relevant  docs 
in  the  cluster;  the  “nrel/all  nrel”  series  means  the  number  of  irrelevant  docs  divide  the  number  of 
all  irrelevant  docs  in  each  cluster.  We  can  discover  that  it’s  not  easy  to  cluster  relevant  and 
irrelevant  documents  into  separate  clusters  by  direct  using  clustering  algorithms. 

3.2  Document  Relevance  Model 

Under  the  assumption  that  relevant  documents  are  more  relevant  to  the  annotated  relevant 
documents  and  less  relevant  to  the  annotated  irrelevant  documents,  we  suppose  that  the  relevance 
measure  between  annotated  documents  and  other  documents  is  useful  for  determining  the 
relevance  between  query  and  documents.  In  this  approach,  the  original  search  result  is  reranked 
based  on  the  relevance  measures  between  the  annotated  documents  (feedback)  and  the  other 
documents  in  the  search  result  list. 

The  score  function  for  one  document  is  as  follows. 

Scoreidi)  =  a  *  norm{max[dis{di,  )]-  max[dis{di,  )]}  —  (1  —  a) 

*  norm{simlarity),  dieD^earc h^^^ore^di) 

D search  is  the  Set  of  Originally  retrieved  documents ;  is  the  feedback  relevant  documents  set; 
D„rei  is  the  feedback  irrelevant  documents  set;  dis(...)  represents  the  cosine  distance  between 
documents;  similarity  means  the  original  score  of  one  document;  norm{. . .)  means  normalization. 
The  training  results  (on  non-  submission  topics)  are  shown  in  the  following  table. 


Table  1 :  MAP  improvement  on  training  set 


RFSet 

RF  doc  count 

Baseline  MAP 

a 

Improved  MAP 

Imp 

B 

1 

0.2940 

0.35 

0.3268 

+11.16% 

C 

6 

0.2860 

0.32 

0.3247 

+13.53% 

D 

10 

0.2805 

0.33 

0.3307 

+17.90% 

E 

246 

0.2122 

0.23 

0.2889 

+36.15% 

From  Table  1,  we  can  see  that  the  MAP  is  steadily  increasing  with  the  increase  of  the  number  of 


RF  documents.  However,  this  isn’t  discovered  in  the  evaluation  results  of  our  submitted  runs, 
which  is  illustrated  in  Section  5. 

4,  Evaluation  Results  of  Submitted  Runs 

Table  2  illustrates  the  evaluation  results  of  Query  Expansion  runs.  Consistent  improvement  has 
been  shown  with  the  increase  of  RF  information  from  RF  set  B  to  D.  However,  the  improvement 
decreases  for  RF  set  E.  This  is  possibly  caused  by  the  irrelevant  documents  in  RF-set  E.  Further 
analysis  will  be  made  when  the  qrels  is  given. 


Table  2:  Evaluation  results  of  Query  Expansion  runs 


runs 

map 

Imp+ 

bpref 

Imp+ 

R-prec 

Imp+ 

Mtc 

Imp+ 

A 

0.1357 

0 

0.1970 

0 

0.1571 

0 

0.0483 

0 

B1 

0.1498 

10.4% 

0.2108 

7.0% 

0.1703 

8.4% 

0.0573 

18.5% 

Cl 

0.1568 

15.5% 

0.2233 

13.4% 

0.1802 

14.7% 

0.0595 

23.1% 

D1 

0.1646 

21.3% 

0.2319 

17.8% 

0.1939 

23.4% 

0.0596 

23.4% 

El 

0.1568 

15.5% 

0.2309 

17.2% 

0.1899 

20.9% 

0.0606 

25.5% 

Table  3  shows  the  evaluation  of  Search  Result  Re-ranking  runs.  Best  MAP  improvement  is 
achieved  On  RF-set  E.  Further  investigation  will  be  given  when  qrels  is  given. 


Table  3 :  Evaluation  results  of  Search  Result  Re-ranking  runs 


runs 

map 

Imp+ 

bpref 

Imp+ 

R-prec 

Imp+ 

mtc 

Imp+ 

A 

0.1357 

0 

0.1970 

0 

0.1571 

0 

0.0483 

0 

B2 

0.1684 

24.1% 

0.2262 

14.8% 

0.1893 

20.5% 

0.0642 

32.9% 

C2 

0.1525 

12.4% 

0.2169 

10.1% 

0.1724 

9.8% 

0.0595 

23.2% 

D2 

0.1607 

18.4% 

0.2262 

14.8% 

0.1859 

18.4% 

0.0598 

23.9% 

E2 

0.1965 

44.8% 

0.2866 

45.5% 

0.2348 

49.5% 

0.0614 

27.1% 

Table  4  shows  the  evaluation  results  of  all  runs. 

Table  4:  Evaluation  results  of  all  runs 


runs 

map 

lmp+ 

R^)^ec 

Imp4- 

mtc 

lmp+ 

A1 

0.1357 

0 

0.1571 

0 

0.0483 

0 

B1 

0.1498 

10.4% 

0.1703 

84% 

0.0573 

18.5% 

B2 

0.1684 

24.1% 

0.1893 

20.5% 

0.0642 

32.9% 

C1 

0.1568 

15.5% 

0.1802 

14.7% 

0.0595 

23.1% 

C2 

0.1525 

12.4% 

0.1724 

9.8% 

0.0595 

23.2% 

D1 

0.1646 

21.3% 

0.1939 

23.4% 

0.0596 

23.4% 

D2 

0.1607 

18.4% 

0.1859 

18.4% 

0.0598 

23.9% 

El 

0.1568 

15.5% 

0.1899 

20.9% 

0.0606 

25.5% 

E2 

0.1965 

44.8% 

0.2348 

49.5% 

0.0614 

27.1% 

Best 

0.4215 

0.4530 

0.0868 

Median 

0.1427 

0.1801 

0.0564 

Worst 

0.0008 

0.0040 

0.0057 

Figure  3  gives  the  per-topic  analysis  on  the  greatest  MAP  improvement  of  Query  Expansion 
approach  (Dl-Al)  and  Search  Result  Re-ranking  approach  (E2-A1).  Both  approaches  are  effective 
on  majority  of  topics.  And  Search  Result  Re-ranking  approach  is  slightly  better  than  the  Query 


Expansion  approach. 


Figure  3:  Improvement  of  D1  vs.  E2  on  every  topic 

5,  Discussion  and  Future  Work 

In  the  Relevance  Feedback  Track,  we  studied  different  approaches  to  improving  retrieval 
performance  using  feedback  information.  Both  the  Query  Expansion  approach  and  the  Search 
Result  Re-ranking  approach  are  effective  for  MAP  improvement  on  majority  of  topics. 

In  the  future,  further  investigation  and  comparative  studied  will  be  made  on  irrelevant  information 
in  different  feedback  sets. 
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